<html>
<head><meta charset="utf-8"><title>mmap concerns · t-compiler/wg-self-profile · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/index.html">t-compiler/wg-self-profile</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html">mmap concerns</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="161852999"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161852999" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161852999">(Mar 27 2019 at 12:15)</a>:</h4>
<blockquote>
<p><span class="user-mention silent" data-user-id="116155">Jake Goulding</span> that's fine in this case, i.e. there's no real risk here</p>
</blockquote>
<p>Why is that?</p>



<a name="161853086"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853086" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853086">(Mar 27 2019 at 12:16)</a>:</h4>
<p>(This is one of my big problems with the mmap crate, which doesn't clearly explain this unsafety in the docs the last time I checked)</p>



<a name="161853092"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853092" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853092">(Mar 27 2019 at 12:16)</a>:</h4>
<p>the file is exclusively accessed by a single process and all data is written just once</p>



<a name="161853110"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853110" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853110">(Mar 27 2019 at 12:17)</a>:</h4>
<p>i.e. we open a new file, touch each byte once for writing and never look at it again</p>



<a name="161853133"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853133" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853133">(Mar 27 2019 at 12:17)</a>:</h4>
<blockquote>
<p>exclusively accessed by a single process</p>
</blockquote>
<p>That's my point though; you literally cannot guarantee that. While it's writing, an automated program can see it and start writing to it as well</p>



<a name="161853189"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853189" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853189">(Mar 27 2019 at 12:18)</a>:</h4>
<p>the same is true for basically everything in the compiler where the file system is touched</p>



<a name="161853208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853208">(Mar 27 2019 at 12:18)</a>:</h4>
<p>Sure, but in the case of <code>File</code>, you don't introduce memory unsafety, the file just gets clobbered</p>



<a name="161853244"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853244" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853244">(Mar 27 2019 at 12:19)</a>:</h4>
<p>do you have a specific scenario in mind that worries you?</p>



<a name="161853322"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853322" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853322">(Mar 27 2019 at 12:20)</a>:</h4>
<p>we are not using the mmap for communicating between threads or processes</p>



<a name="161853351"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853351" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853351">(Mar 27 2019 at 12:21)</a>:</h4>
<p>it is just supposed to let the OS do some paging magic</p>



<a name="161853354"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853354" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853354">(Mar 27 2019 at 12:21)</a>:</h4>
<p>With file-backed-mmap, you get a <code>*mut T</code> and convert it to a <code>&amp;T</code> or <code>&amp;mut T</code>. During the time that reference exists, if anything else decides to write to that file, the reference's guarantees are invalidated.</p>



<a name="161853448"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853448" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853448">(Mar 27 2019 at 12:22)</a>:</h4>
<p>I do not have any specific scenarios, no. I'm just pointing out that, as far I have been able to determine, file-backed-mmap cannot be used in Rust without introducing memory unsafety based on my understanding of the rules.</p>



<a name="161853477"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853477" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853477">(Mar 27 2019 at 12:22)</a>:</h4>
<p>It doesn't matter what the intended use is, only what the <em>possible</em> uses are.</p>



<a name="161853480"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853480" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853480">(Mar 27 2019 at 12:22)</a>:</h4>
<p>well yeah, the memory/file is accessed only from unsafe code anyway</p>



<a name="161853555"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853555" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853555">(Mar 27 2019 at 12:23)</a>:</h4>
<p>but that's like saying you can't use <code>&amp;mut []</code> safely because someone else might use <code>unsafe</code> to get another reference</p>



<a name="161853610"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853610" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853610">(Mar 27 2019 at 12:24)</a>:</h4>
<blockquote>
<p>only from unsafe code anyway</p>
</blockquote>
<p>You aren't allowed to violate Rust's guarantees in an <code>unsafe</code> block. The compiler just cannot validate them, so it's up to the programmer.</p>



<a name="161853619"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853619" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853619">(Mar 27 2019 at 12:24)</a>:</h4>
<blockquote>
<p>but that's like saying you can't use <code>&amp;mut []</code> safely because someone else might use <code>unsafe</code> to get another reference</p>
</blockquote>
<p>I'm sorry if I've given this impression, but it's definitely not the same case.</p>



<a name="161853653"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853653" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853653">(Mar 27 2019 at 12:25)</a>:</h4>
<p>we are doing a runtime check that makes sure that we don't violate the invariants in the current process</p>



<a name="161853665"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853665" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853665">(Mar 27 2019 at 12:25)</a>:</h4>
<p>In the <code>&amp;mut[]</code> case, the person who wrote the <code>unsafe</code> block code didn't uphold the reference's guarantees of non-mutability / single-mutable</p>



<a name="161853675"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853675" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853675">(Mar 27 2019 at 12:25)</a>:</h4>
<p>if another process writes over the file, well that's a bug</p>



<a name="161853702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853702">(Mar 27 2019 at 12:25)</a>:</h4>
<p>A bug that (I believe) introduces memory unsafety</p>



<a name="161853761"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853761" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853761">(Mar 27 2019 at 12:26)</a>:</h4>
<p>Thus my statement: </p>
<blockquote>
<p>file-backed-mmap cannot be used in Rust without introducing memory unsafety</p>
</blockquote>



<a name="161853765"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853765" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853765">(Mar 27 2019 at 12:26)</a>:</h4>
<p>so you are worried about an attacker?</p>



<a name="161853796"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853796" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Hanna Kruppe <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853796">(Mar 27 2019 at 12:26)</a>:</h4>
<p>Chiming in from the back row to say that you can side step these concerns by creating a <code>&amp;[Cell&lt;u8&gt;]</code> and never having a reference to a non-cell type in the first place</p>



<a name="161853802"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853802" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853802">(Mar 27 2019 at 12:26)</a>:</h4>
<p>if no other process access the file, then no assumptions that the compiler made are violated</p>



<a name="161853831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853831">(Mar 27 2019 at 12:27)</a>:</h4>
<p>There's two schools of thought about that: does UB occur only when it happens, or when it <em>could</em> happen? I believe Rust falls into the latter case.</p>



<a name="161853915"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853915" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853915">(Mar 27 2019 at 12:28)</a>:</h4>
<p>I thought the problem with UB is that the compiler does invalid optimizations</p>



<a name="161853934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161853934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161853934">(Mar 27 2019 at 12:28)</a>:</h4>
<p><span class="user-mention" data-user-id="124289">@rkruppe</span> ah, that's a missing piece I hadn't considered. I wonder how painful it is to read/write to such a thing</p>



<a name="161854136"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161854136" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161854136">(Mar 27 2019 at 12:31)</a>:</h4>
<p><span class="user-mention" data-user-id="124287">@mw</span>  for example</p>
<div class="codehilite"><pre><span></span><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">42</span><span class="p">;</span><span class="w"></span>
<span class="w">    </span><span class="k">unsafe</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">a</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">_</span><span class="p">);</span><span class="w"></span>
<span class="w">        </span><span class="kd">let</span><span class="w"> </span><span class="k">mut</span><span class="w"> </span><span class="n">b</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="o">*</span><span class="p">(</span><span class="o">&amp;</span><span class="k">mut</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">_</span><span class="p">);</span><span class="w"></span>

<span class="w">        </span><span class="n">println</span><span class="o">!</span><span class="p">(</span><span class="s">&quot;{:p}, {:p}&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="n">b</span><span class="p">);</span><span class="w"></span>
<span class="w">    </span><span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>Miri reports this as causing undefined behavior. We never actually mutate anything; the sheer existence of the aliased pointers is enough.</p>



<a name="161854479"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161854479" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161854479">(Mar 27 2019 at 12:36)</a>:</h4>
<p>I don't see how this relates to memory mapped files that are accessed exclusively by one process</p>



<a name="161855099"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855099" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Hanna Kruppe <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855099">(Mar 27 2019 at 12:45)</a>:</h4>
<p>The caveat "exclusively accessed by one process" is a big one because that's not something you can do in that process to ensure others won't access the file, and unlike e.g. unsafe code in the same process duplicating mutable references, there is no convention or assignment of responsibility that would prevent it. Perfectly well-behaved processes and well-intentioned users can trample all over your mmap'd file. IMO a program that "only has UB if other processes access the file" is just as buggy as a program that "only has UB if the input file contains &lt;perfectly plausible errors&gt;", e.g. a malformed gzip archive that leads to a buffer overflow in a decompression program.</p>



<a name="161855180"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855180" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855180">(Mar 27 2019 at 12:46)</a>:</h4>
<p>That is a wonderfully better way of expressing my concerns, thank you :-)</p>



<a name="161855250"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855250" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855250">(Mar 27 2019 at 12:47)</a>:</h4>
<p>Practically, I don't know if anything bad will happen <em>now</em>. What's most insidious is when compiler changes in the future break seemingly "working" code.</p>



<a name="161855549"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855549" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855549">(Mar 27 2019 at 12:51)</a>:</h4>
<p>I'm still not convinced this applies here :)</p>



<a name="161855570"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855570" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855570">(Mar 27 2019 at 12:51)</a>:</h4>
<p>I'm not trying to convince you that mutable shared memory maps are fine in general</p>



<a name="161855584"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855584" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855584">(Mar 27 2019 at 12:51)</a>:</h4>
<p>I totally see what your concern is</p>



<a name="161855676"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855676" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855676">(Mar 27 2019 at 12:52)</a>:</h4>
<p>I just don't see a real risk in this specific case</p>



<a name="161855894"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855894" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855894">(Mar 27 2019 at 12:55)</a>:</h4>
<p>I mean by that same argument, we would not be allowed to read crate metadata from disk because someone could have changed the length of an array in the binary format</p>



<a name="161855971"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161855971" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161855971">(Mar 27 2019 at 12:56)</a>:</h4>
<p>this is a form of FFI where there is always some risk that some code outside of Rust has changed something unexpectedly</p>



<a name="161856434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161856434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161856434">(Mar 27 2019 at 13:02)</a>:</h4>
<blockquote>
<p>I mean by that same argument, we would not be allowed to read crate metadata from disk because someone could have changed the length of an array in the binary format</p>
</blockquote>
<p>I'm not sure this is the same argument. In that case, with an untrusted input like the file, we would perform some check in our process that the array length is correct. If it wasn't, we don't allow access to that data (panic, return a <code>Result</code>, etc.). </p>
<p>If we <em>didn't</em> perform that check, but assumed that the array length was accurate and assumed that data at some offset was of a specific form, then that might lead to memory unsafety and should indeed be disallowed.</p>



<a name="161856813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161856813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161856813">(Mar 27 2019 at 13:06)</a>:</h4>
<p>With file-backed-mmap, there's no possible check that our program can make to know that the underlying data hasn't changed.</p>



<a name="161856989"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161856989" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161856989">(Mar 27 2019 at 13:08)</a>:</h4>
<p>but the data is in the file is not read by the rustc process, it is only written</p>



<a name="161857004"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161857004" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Hanna Kruppe <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161857004">(Mar 27 2019 at 13:08)</a>:</h4>
<p>I personally don't care super strongly about this situation :) But the one thing that actually gives me a stomach ache when mmap'd files come up, is: what precedent does this set? Even if in this situation the risk is tiny and acceptable for big performance wins, I worry that there is insufficient caution about the pitfalls of mmap in the community. For example, rustc's clearly not intended to be run in a hostile execution environment, and that's fine for rustc, but a different project may have different needs and if its authors are unaware of the potential UB of mmap, they may handle it insufficiently.</p>



<a name="161857176"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161857176" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161857176">(Mar 27 2019 at 13:10)</a>:</h4>
<p>that's something that should be documented in the <code>memmap</code> crate, I think</p>



<a name="161857227"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161857227" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161857227">(Mar 27 2019 at 13:11)</a>:</h4>
<p>I mean, I don't mind adding a big warning comment to the measureme crate too, should it turn out that mmap files are actually faster</p>



<a name="161857319"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161857319" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161857319">(Mar 27 2019 at 13:12)</a>:</h4>
<p>but memory mapped files are a fundamental OS facility, they should be usable in Rust</p>



<a name="161857726"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161857726" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161857726">(Mar 27 2019 at 13:17)</a>:</h4>
<blockquote>
<p>that's something that should be documented in the <code>memmap</code> crate, I think</p>
</blockquote>
<p>I agree, which is why I started this thread with "(This is one of my big problems with the mmap crate, which doesn't clearly explain this unsafety in the docs the last time I checked)"</p>



<a name="161857885"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161857885" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161857885">(Mar 27 2019 at 13:19)</a>:</h4>
<p>Although I think that the crate itself is flawed, as it gives you back a type which derefs to <code>[u8]</code> so you can't even use a <code>Cell</code>, it's already too late.</p>



<a name="161857975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161857975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161857975">(Mar 27 2019 at 13:20)</a>:</h4>
<blockquote>
<p>I mean, I don't mind adding a big warning comment to the measureme crate too, should it turn out that mmap files are actually faster</p>
</blockquote>
<p>The problem is that the warning is unactionable. It would be like "if you use mmap, then your program has undefined behavior, but it's faster!". What do I do with that as a user?</p>



<a name="161858123"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161858123" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161858123">(Mar 27 2019 at 13:22)</a>:</h4>
<p>the warning would be "we carefully checked that this specific use of memory mapped files does not introduce undefined behaviour. that does not mean that memory mapped files are easy to use from Rust"</p>



<a name="161858145"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161858145" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161858145">(Mar 27 2019 at 13:22)</a>:</h4>
<p>that's rather different, I'd say</p>



<a name="161858748"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161858748" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161858748">(Mar 27 2019 at 13:30)</a>:</h4>
<p>I'm afraid that I'm saying the same thing over and over, so I apologize. I can see that I'm not going to be able to convince you, so perhaps I'm wrong. My last attempt:</p>
<blockquote>
<p>this specific use of memory mapped files does not introduce undefined behaviour</p>
</blockquote>
<p>There is no way we can make such a statement. The running program is completely unable to make this guarantee.</p>



<a name="161859087"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161859087" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161859087">(Mar 27 2019 at 13:34)</a>:</h4>
<p>even if it nevers casts a pointer into the memory in question to a <code>&amp;</code> or <code>&amp;mut</code>? i.e. if it only ever uses raw pointers to that memory?</p>



<a name="161859095"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161859095" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Wesley Wiser <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161859095">(Mar 27 2019 at 13:34)</a>:</h4>
<p>On the one hand, I agree: using <code>mmap</code> does seem to create the potential to violate Rust's semantics and introduce UB. On the other hand, if the scenario we're worried about is another process causing the violation to happen, I'm not sure why this is an issue and <code>ptrace</code> isn't. Ultimately we're at the mercy of the OS not to do something which violates the language semantics.</p>



<a name="161859344"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161859344" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161859344">(Mar 27 2019 at 13:37)</a>:</h4>
<p>i.e. if I only ever use raw pointers, there should not be any UB, right? same as with <code>Cell</code></p>



<a name="161859899"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161859899" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161859899">(Mar 27 2019 at 13:43)</a>:</h4>
<blockquote>
<p>That's my point though; you literally cannot guarantee that. While it's writing, an automated program can see it and start writing to it as well</p>
</blockquote>
<p>You can lock the file.</p>



<a name="161860131"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161860131" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161860131">(Mar 27 2019 at 13:45)</a>:</h4>
<p>The API you’re looking for is <code>flock</code> that could maintain the invariant of 1 writer 0 readers or at least 1 writer many readers.</p>



<a name="161860268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161860268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161860268">(Mar 27 2019 at 13:46)</a>:</h4>
<p>then the next complaint will be that these locks are only advisory and can be ignored at will :P</p>



<a name="161860845"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161860845" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161860845">(Mar 27 2019 at 13:53)</a>:</h4>
<p>… Sure.</p>



<a name="161862305"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161862305" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161862305">(Mar 27 2019 at 14:08)</a>:</h4>
<p>As I mentioned in some issue before, you could use <code>pwrite</code> and <code>pread</code> to write and read at arbitrary offsets in a file.</p>



<a name="161862343"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161862343" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161862343">(Mar 27 2019 at 14:09)</a>:</h4>
<p>You would have to get rid of conveniences that mmap affords you, but functionally there should be little difference.</p>



<a name="161863616"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/mmap%20concerns/near/161863616" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mw <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/mmap.20concerns.html#161863616">(Mar 27 2019 at 14:22)</a>:</h4>
<p>these sound expensive, being system calls?</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>